- Selecting a dataset
- Exploring the dataset to identify what kinds of questions can be answered using the dataset
- Identifying one research question
- Using pandas methods to explore the dataset - this also involves using visualization techniques using matplotlib
- Reporting findings/analyses
- Presenting the work in the given presentation template
The dataset is available for download here - https://grouplens.org/datasets/movielens/20m/
Description about the dataset, as shown on the website is below:
This dataset (ml-20m) describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 20000263 ratings and 465564 tag applications across 27278 movies. These data were created by 138493 users between January 09, 1995 and March 31, 2015. This dataset was generated on October 17, 2016.
Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.
The data are contained in six files, genome-scores.csv, genome-tags.csv, links.csv, movies.csv, ratings.csv and tags.csv. More details about the contents and use of all these files follows.
This and other GroupLens data sets are publicly available for download at http://grouplens.org/datasets/.
In [199]:
# The first step is to import the dataset into a pandas dataframe.
import pandas as pd
#path = 'C:/Users/hrao/Documents/Personal/HK/Python/ml-20m/ml-20m/'
path = '/Users/Harish/Documents/HK_Work/Python/ml-20m/'
movies = pd.read_csv(path+'movies.csv')
movies.shape
Out[199]:
In [200]:
tags = pd.read_csv(path+'tags.csv')
tags.shape
Out[200]:
In [201]:
ratings = pd.read_csv(path+'ratings.csv')
ratings.shape
Out[201]:
In [202]:
links = pd.read_csv(path+'links.csv')
links.shape
Out[202]:
In [203]:
movies.head()
Out[203]:
In [204]:
tags.head()
Out[204]:
In [205]:
ratings.head()
Out[205]:
In [206]:
links.head()
Out[206]:
- Is there a correlation or a trend between the year of release of a movie and the genre?
- Which genres were more dominant in each decade of the range available in the dataset?
- Do science fiction movies tend to be rated more highly than other movie genres?
In [207]:
# List of genres as a Python list
genres = ['Action','Adventure','Animation','Children','Comedy','Crime','Documentary','Drama','Fantasy','Film-Noir','Horror','Musical','Mystery','Romance','Sci-Fi','Thriller','War','Western']
In [208]:
genres_rating_list = []
In [209]:
# The loop reads each element of the above list
# For each iteration, one genre is selected from the movies data frame
# This selection of the data frame is then merged with the rating data frame to get the rating for that genre
# Once the new merged data frame is created, we use the mean function to get the mean rating for the genre
# The genre and the corresponding mean rating are then appended to the genres_rating Data Frame
# The entire looping takes long - can certainly be optimized for performance
for i in range(len(genres)):
fil = genres[i]+'_filter'
mov = genres[i]+'_movies'
rat = genres[i]+'_ratings'
rat_mean = rat+'_mean'
fil = movies['genres'].str.contains(genres[i])
mov = movies[fil]
rat = mov.merge(ratings, on='movieId', how='inner')
rat_mean = round(rat['rating'].mean(), 2)
#print(genres[i], round(rat_mean,2))
genres_rating_list.append(rat_mean)
In [210]:
df = {'Genre':genres, 'Genres Mean Rating':genres_rating_list}
In [211]:
genres_rating = pd.DataFrame(df)
In [212]:
genres_rating
Out[212]:
In [213]:
genres_rating['Genres Standard Deviation'] = genres_rating['Genres Mean Rating'].std()
In [214]:
genres_rating['Mean'] = genres_rating['Genres Mean Rating'].mean()
genres_rating['Zero'] = 0
In [215]:
genres_rating
Out[215]:
In [216]:
overall_mean = round(genres_rating['Genres Mean Rating'].mean(), 2)
overall_std = round(genres_rating['Genres Mean Rating'].std(),2)
scifi_rating = genres_rating[genres_rating['Genre'] == 'Sci-Fi']['Genres Mean Rating']
In [217]:
print(overall_mean)
In [218]:
print(overall_std)
In [219]:
print(scifi_rating)
In [220]:
genres_rating['Diff from Mean'] = genres_rating['Genres Mean Rating'] - overall_mean
In [221]:
genres_rating
Out[221]:
In [222]:
genre_list = list(genres_rating['Genre'])
In [223]:
genres_rating_list = list(genres_rating['Genres Mean Rating'])
genres_diff_list = list(genres_rating['Diff from Mean'])
In [224]:
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 10))
ax1 = plt.subplot(2,1,1)
x = [x for x in range(0, 18)]
xticks_genre_list = genre_list
y = genres_rating_list
plt.xticks(range(len(x)), xticks_genre_list)
plt.scatter(x,y, color='g')
plt.plot(x, genres_rating['Mean'], color="red")
plt.autoscale(tight=True)
#plt.rcParams["figure.figsize"] = (10,2)
plt.title('Movie ratings by genre')
plt.xlabel('Genre')
plt.ylabel('Rating')
plt.ylim(ymax = 4, ymin = 3)
plt.grid(True)
plt.savefig(r'movie-ratings-by-genre.png')
plt.annotate("Sci-Fi Rating",
xy=(14.25,3.5), xycoords='data',
xytext=(14.20, 3.7), textcoords='data',
arrowprops=dict(arrowstyle="->",
connectionstyle="arc3"),
)
for i,j in enumerate( y ):
ax1.annotate( j, ( x[i] + 0.03, y[i] + 0.02))
ax2 = plt.subplot(2,1,2)
x = [x for x in range(0, 18)]
xticks_genre_list = genre_list
y = genres_rating['Diff from Mean']
plt.xticks(range(len(x)), xticks_genre_list)
plt.plot(x,y)
plt.plot(x, genres_rating['Zero'])
plt.autoscale(tight=True)
#plt.rcParams["figure.figsize"] = (10,2)
plt.title('Deviation of each genre\'s rating from the overall mean rating')
plt.xlabel('Genre')
plt.ylabel('Deviation from mean rating')
plt.grid(True)
plt.savefig(r'deviation-from-mean-rating.png')
plt.annotate("Sci-Fi Rating",
xy=(14,-0.13), xycoords='data',
xytext=(14.00, 0.0), textcoords='data',
arrowprops=dict(arrowstyle="->",
connectionstyle="arc3"),
)
plt.show()
The scatter plot shows the mean rating value for each genre. Each genre has a value on the scatter plot for the mean rating value for that genre. Let us now see if the plot is able to help us answer the question above.
The mean rating for Sci-Fi genre is about 3.45. When looking at the plot, we see that there are only three other genres out of 18 genres in total, that have lesser mean ratings than Sci-Fi - Horror, Children and Comedy. The remaining 10 genres have mean ratings higher than Science Fiction.
This gives us enough information to answer the question. Sci-Fi movies do not tend to be rated higher than other genres.
The second plot, a bar plot, shows how much each genre's ratings deviate from the overall mean of ratings. Science Fiction is around -0.13 lower than the mean rating of 3.58, showing lesser deviation than Horror at the lower end and Film-Noir at the higher end.
- Is there a correlation or a trend between the year of release of a movie and the genre?
- Which genres were more dominant in each decade of the range available in the dataset?
In [225]:
# extract year of release of each movie from the title column
# convert the data type of the movie_year column to numeric (from str)
import numpy as np
import re
#movies['rel_year'] = movies.title.str[-5:-1]
#movies['rel_year'] = movies.title.apply(lambda x: x[-5:-1])
movies['movie_year'] = movies['title']
#movies['movie_year'] = movies['movie_year'].apply(lambda x: re.findall('\((.*?)\)',x))
movies['movie_year'] = movies['movie_year'].str.extract(r"\(([0-9]+)\)", expand=False)
# creating a new column with just the movie titles
movies['title_only'] = movies['title']
movies['title_only'] = movies['title_only'].str.extract('(.*?)\s*\(', expand=False)
In [234]:
#Drop all rows containing incorrect year values - such as 0, 6, 69, 500 and -2147483648
movies.drop(movies[movies.movie_year == '0'].index, inplace=True)
movies.drop(movies[movies.movie_year == '6'].index, inplace=True)
movies.drop(movies[movies.movie_year == '06'].index, inplace=True)
movies.drop(movies[movies.movie_year == '69'].index, inplace=True)
movies.drop(movies[movies.movie_year == '500'].index, inplace=True)
movies.drop(movies[movies.movie_year == '-2147483648'].index, inplace=True)
In [235]:
movies['movie_year'].fillna(0, inplace=True)
In [236]:
#convert the string values to numeric
movies['movie_year'] = pd.to_datetime(movies['movie_year'], format='%Y')
Now that we have a move year column, let us list the data types of the columns in the movies data frame.
In [237]:
movies.dtypes
Out[237]:
movie_year is of float64 datat type. We must convert the data type of the movie_year column to int64. Before we go ahead and do that, we must replace all NULL and inifinite entries in the column with zero. If we do not perform this step, we will get the following errror message.
In [188]:
#movies['movie_year'].astype(np.int64)
In [190]:
#movies['movie_year'] = movies['movie_year'].astype(np.int64, inplace=True)
In [191]:
movies.dtypes
Out[191]:
In [253]:
movies.describe()
Out[253]:
In [254]:
movies.head()
Out[254]:
In [290]:
movies_and_years = pd.DataFrame(movies['movie_year', 'title_only'])
In [289]:
movies_and_years.tail()
Out[289]:
In [284]:
#a = pd.DataFrame(movies['title_only'].groupby(movies['movie_year']).count(), movies['movie_year'], movies['title_only'])
In [285]:
#plt.plot(a['title_only'], a['movie_year'])
#plt.show()
In [250]:
#movies_and_years
In [251]:
#movies_and_years['title_only'].groupby[movies_and_years['movie_year']]
In [ ]:
#create a unique array of years from the newly created data frame column
#years_array_raw = movies['movie_year'].unique()
#years_array_raw = years_array_raw.astype(int)
In [ ]:
#years_array_raw
In the above array, we see some incorrect year values. And the year values need to be sorted as well.
In [ ]:
# We see some elements of the array that are not year values - such as -2147483648 and 6
#Let us create a new numpy array to clean up the data and sort the list of unique years
In [ ]:
# Numpy arrays are immutable. Delete operations on the array will create a new copy of the same array. So,
#years_array_clean = np.array([])
In [ ]:
#for i in range(len(years_array_raw)):
# if years_array_raw[i] <= 2015 and years_array_raw[i] >= 1800:
# years_array_clean = np.append(years_array_clean, years_array_raw[i])
# else:
# print(years_array_raw[i], ' - Incorrect year value. Will not be appended to the new array')
In [ ]:
#the incorrect year values are not present in the array anymore
#years_array_clean
In [ ]:
#sorting the years in ascending order
#years_array_clean = np.sort(years_array_clean)
In [ ]:
#Now, we have a unique, clean and sorted year list
#years_array_clean
In [245]:
# years and their respective movie counts
#movies_and_years = pd.DataFrame(movies['movie_year'],(movies['title_only'].groupby(movies['movie_year']).agg('count')))
#movies_and_years1 = pd.DataFrame(movies['movie_year'],(movies['title_only'].groupby(years_array_clean)))
In [ ]:
# Movies and their corresponding genre combinations
# This is not the ideal grouping I want to arrive at - I want to analyze movies by individual genres, not groups of genres such as Comedy|Drama|Romance, for example.
# That will be the next step, possibly by creating new columns with flags for each genre
#movies_and_genres = pd.DataFrame(movies['genres'], (movies['title_only'].groupby(movies['genres']).agg('count')))